1. Field of the Invention
The present invention relates to a method for visual servoing or aligning of apparatus and, more specifically, to a method of visual servoing of a linear apparatus rotatable in a plane about a fixed point.
2. Discussion of the Related Art
Linear apparatus comprise linear, lineal or line shaped apparatus and apparatus whereof at least a portion comprises a linear, line shaped part or which exhibits some lineal or line-shaped characteristic, such as, by way of example, a rod, a needle, a linear industrial tool such as a drill, a reamer, a probe, a drafting type of instrument, optical or microscope adjunct equipment, or generally any implement that has at least a portion that exhibits a line-like silhouette or outline or which has a characteristic that can be represented by a straight line, or two parallel straight lines, or generally by any implement that includes at least a portion that exhibits a cylindrical or near-cylindrical shape.
Generally, prior art servoing methods comprise making iterative corrections of the 3-dimensional (3D) position of all object based on visual feedback from the image. Whether such methods are performed manually or automatically, they have in common the fact that the needle is moved in 3D in a trial and error fashion until the projection of the needle is visually aligned to the projection of the target in the image. The number of iterations required for such an approach to converge is not known a priori and can be fairly high: 10-50 or more iterations may typically be needed.
Therefore, a need exists for an improved system and method is needed for servoing the linear apparatus.
Applications involving alignment include the automatic placement of needles for percutaneous procedures and biopsies, as indicated in FIG. 3. For example, needle placement can be achieved by visual servoing of the needle in several successive planes, until the needle is in the desired position and orientation to reach a particular anatomical site in a patient. The action is performed under X-ray fluoroscopy before insertion of the needle. A detailed description of the method is described in Appendix A. The principles of the present invention are applicable to such visual servoing.
Also of interest to the subject matter of the present invention are the following: copending U.S. patent application Ser. No. 08/935,269, entitled APPARATUS AND METHOD FOR DETECTION AND LOCALIZATION OF A BIOPSY NEEDLE OR SIMILAR SURGICAL TOOL IN A RADIOGRAPHIC IMAGE, with a filing date of Sep. 22, 1997, filed in the name of Navab; copending U.S. patent application Ser. No. 08/722,707 for N. Navab et al., entitled APPARATUS AND METHOD FOR AUTOMATICALLY POSITIONING A BIOPSY NEEDLE; copending U.S. patent application Ser. No. 08/722,708 for N. Navab et al., entitled APPARATUS AND METHOD FOR DETERMINING THE CORRECT INSERTION DEPTH FOR A BIOPSY NEEDLE; copending U.S. patent application Ser. No. 08/722,724 for N. Navab et al., entitled TRIGONONMETRIC DEPTH GAUGE FOR BIOPSY NEEDLE; copending U.S. patent application Ser. No. 08/935,270 for N. Navab et al., entitled METHOD FOR LOCALIZATION OF A BIOPSY NEEDLE OR SIMILAR SURGICAL TOOL IN A RADIOGRAPHIC IMAGE; copending U.S. patent application Ser. No. 08/107,787 for N. Navab et al., entitled METHOD AND APPARATUS FOR PRECISE NEEDLE PLACEMENT USING VISUAL SERVOING. The disclosure of the foregoing is herein incorporated by reference to the extent not incompatible with the present invention.
It is herein recognized that visual aiming may also include applications in surveillance or process supervision where the goal can be to keep an object in the center of, for example, a camera field of view or in a location for facilitating the fixing of attention on the object. Other applications include robot guidance in which robot motion is effected and controlled for obtaining alignment of a robot arm or manipulator arm portion with a target.
Linear apparatus of the type to which the method of the present invention applies is rotatable about a defined center point of rotation and may rotatably pivoted about such a point of rotation. Furthermore, such rotation takes place in a plane including within itself the point of rotation.
In practicing the method of the present invention, the linear apparatus is observed through an imaging device, which could be a camera utilizing a charge coupled imaging device (CCD) in combination with suitable optical components, an )X-ray fluoroscope or some other suitable kind of imaging device.
The imaging device produces an image of the linear apparatus and of its surrounding environment. The image is conveniently stored in a memory in digitally encoded format which also facilitates further processing of the image, and may be displayed in a convenient form such as on a screen as commonly used in conjunction with a fluoroscope or otherwise as may be convenient. In accordance with the invention, the linear device is visually aligned with a defined target point which is also defined in the image. Such a target point may be defined interactively by the user of the present method or it may be found automatically using image processing techniques.
An aspect of the present invention is that it performs visual alignment of the linear apparatus to a target T, that is, it ensures that the linear apparatus appears in an image to be aligned to target T or, in other words, that the projection of the linear apparatus in the image is aligned to the projection of the target in the image. While visual alignment or visual servoing in accordance with this aspect of the invention is not of itself a full 3D alignment of the linear apparatus to target T, nevertheless, for many applications, achieving visual alignment in this sense is sufficient and full 3D alignment may not be required. There are applications wherein the constellation or relative positions of a linear apparatus and target are constrained in ways which are advantageous for the application of visual servoing such that a single performance: of visual servoing provides, in effect, a complete 3D alignment. An example of this is provided by the embodiment wherein labels are affixed to bottles, as will be hereinafter described.
In order to achieve visual servoing, the position of the target in three-dimensional space (xe2x80x9c3D positionxe2x80x9d) need not be known. All that is needed is the position of the 2-dimensional projection of the 3-D target in the image (xe2x80x9c2-D targetxe2x80x9d).
In accordance with a further aspect of the invention, achieving visual alignment is utilized as an intermediate step to achieving full 3D alignment. An object of the present invention is accordingly to provide 3D alignment utilizing the method of visual servoing herein disclosed as an advantageous step in a method for achieving 3D alignment. The afore-cited U.S. patent application Ser. No. 08/722,725 discloses that visual servoing in several successive planes achieves complete 3D alignment, utilizing other methods to perform the visual servoing.
In accordance with an aspect of the present inventions, 3D alignment is achieved through application of visual servoing which utilizes fewer iteration; than was typical heretofore.
In accordance with an aspect of the present inventions, 3D alignment is achieved application of visual servoing which utilizes 3 iterations.
Performing visual servoing in accordance with the method of the present invention comprises positioning the linear apparatus so that it is visually aligned to the target point in the image. See FIG. 1. In accordance with the principles of the present invention, a method and an apparatus for visual servoing of a linear apparatus utilizes just 3 iterations and no calibration of the camera or imaging device is needed.
In accordance with an aspect of the invention a method and apparatus for visual servoing of a linear apparatus is based on the use of cross-ratios, which is a known invariant of projective geometry.
In accordance with an aspect of the invention, a method for visual servoing of a linear apparatus for aiming at a fixed target, the linear apparatus being rotatable in a defined plane about a fixed point, and wherein the linear apparatus and target are viewed by an imaging process in a two-dimensional (2D) image plane, comprises the steps of: beginning at an arbitrary first line direction in space of the linear apparatus, detecting and storing an image of the first line direction in the 2D image plane; detecting and storing an image of the fixed target in the 2D image plane; rotating the: linear apparatus through a first angle in the defined plane about the fixed point to a second line direction; detecting and storing an image of the second line direction in the 2D image plane; rotating the linear apparatus through a second angle in the defined plane; about the fixed point to a third line direction; detecting and storing an image of the third line direction in the 2D image plane; determining and storing the point of intersection in the 2D image plane of the images of the first, second, and third line directions; storing an image of a virtual straight line in the 2D image plane passing through the image of the fixed target in the 2D image plane and the point of intersection, the image of a virtual straight line being associated with a corresponding virtual line passing through the fixed point and the target; determining the cross-ratio of the 2D images of the first, second, third, and virtual line directions; calculating from the cross-ratio an aiming angle between the third and the virtual line directions in 3D space; and aligning the linear apparatus along the virtual line in accordance with the aiming angle so as to visually aim at the target.
In accordance with another aspect of the invention, a method for visual servoing of a linear apparatus includes a step of setting the cross-ratio of the 2D images of the first, second, third, and virtual line directions as equal to the cross-ratio of the 3D first, second, third, and virtual line directions. In accordance with another aspect of the invention, in a method for visual servoing of a linear apparatus, the step of determining the cross-ratio of that images of the first, second, third and virtual line directions comprises the steps of adding an arbitrary virtual line d that intersects the images of the first, second, third, and virtual line directions, hereinafter referred to as l1,l2,l3 and lt determining the cross-ratio c=(l1,l2,l3,lt)=(q1,q2, q3,qt)=(q1q3*q2qt)+(q1qt*q2q3), where q1=l1∩d,q2=l2∩d,q3=l3∩d and qt=l1∩d are the intersections of l1,l2,l3,lt with d.
In accordance with another aspect of the invention, in a method for visual servoing of a linear apparatus, the first and second angles, and the aiming angle, hereinafter referred to as xcex81,xcex82 and xcex8t respectively, and wherein c is the previously determined cross-ratio, and wherein the step of determining the cross-ratio of the 2D images of the first, second, third, and virtual line directions comprises the step of solving an equation for obtaining the aiming angle xcex8t:                     (                  c          -          1                )            ⁢      sin      ⁢              xe2x80x83            ⁢              θ        2            ⁢      cos      ⁢              xe2x80x83            ⁢              θ        t              +                  (                                            c              ⁢                              xe2x80x83                            ⁢              sin              ⁢                              xe2x80x83                            ⁢                              θ                2                                                    tan              ⁢                              (                                                      θ                    1                                    +                                      θ                    2                                                  )                                              -                      cos            ⁢                          xe2x80x83                        ⁢                          θ              2                                      )            ⁢      sin      ⁢              xe2x80x83            ⁢              θ        t              =  0
In accordance with another aspect of the invention, a method for visual servoing of a linear apparatus the steps are performed, at least in part, automatically under programmed computer control.
In accordance with a further aspect of the invention, apparatus for visual servoing of a linear apparatus for aiming at a fixed target, the linear apparatus being rotatable in a defined plane about a fixed point, and wherein the linear apparatus and target are viewed by an imaging process in a two-dimensional (2D) image plane, the apparatus for visual servoing includes computerized apparatus for reading, storing and processing data, and wherein the computerized apparatus detects and stores an image of the fixed target in the 2D image plane; detects and stores an image in the 2D image plane of a first arbitrary line direction in space of the linear apparatus; detects and stores an image in the 2D image plane of a second line direction in space of the linear apparatus wherein the second direction line in space is at a first angle in the defined plane relative to the first line direction; detects and stores an image in the 2D image plane of a third line direction in space of the linear apparatus wherein the third direction line .n space is at a second angle in the defined plane relative to the second line direction; determines and stores the point of intersection in the 2D image plane of the images of the first, second, and third line directions; stores an image of a virtual straight line in the 2D image plane passing through the image of the fixed target in the 2D image plane and the point of intersection, the image of a virtual straight line being associated with a corresponding virtual line passing through the fixed point and the target; determines the cross-ratio of the images of the first, second, third, and virtual line directions; and obtains from the cross-ratio an aiming angle between the third and the virtual line directions for alignment of the linear apparatus along the virtual line in accordance with the aiming angle.
In accordance with still a further aspect of the invention, a method and system or apparatus is disclosed for visual servoing of a linear apparatus rotatable in a plane about a fixed point. The linear apparatus may be, for example, a biopsy needle. The method is based on an application of cross-ratios and provides alignment of the linear apparatus to the target in an image, such as a fluoroscope image in 3 iterations. The imaging device through which the scene is observed can be of any suitable type and does not need to be calibrated.